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Review of Web-Based Technical Documentation Processes 
FY07 NAEP-QA Special Study Report 
Background 

NCES provides detailed technical documentation reports that cover all aspects of the 
NAEP assessments. Because of the breadth and depth of the technical documentation, the 
development and review cycle has become quite long. Detailed technical documentation is not 
widely available to researchers until many years after assessment data have been collected and 
results have been released. This time lag results in an increased burden on NCES and contractor 
staff, who must answer individual technical questions that arise, often repeatedly, before the 
documentation is available. 

Beginning with the 2000 and 2001 NAEP assessments, NCES has made technical 
documentation available on the worldwide web at http ://nces .ed. gov/nationsreportcard/tdw/ . The 
web-based documentation is designed to be less dense and more accessible than prior printed 
versions. The current study reviewed the web-based version of NAEP technical documentation, 
compared it to prior hardcopy versions, and identified possible improvements to both the content 
and development processes used for the web-based documentation. 



Study Questions 

The study addressed three specific questions about the new web-based version of NAEP 
technical documentation. The study questions were: 

1 . How does the new web-based technical documentation for NAEP assessments 
compare with previous hardcopy versions with respect to comprehensiveness 
and accessibility? What advantages and disadvantages may be realized via the 
web-based approach? 

2. To what extent have changes to the development and review processes for 
technical documentation led to improved timeliness? 

3. How will updates to the web-based documentation be handled? Will changes be 
clear to users who may be familiar with previous versions? 
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Approach 

The study was conducted in three phases. The first phase included collecting information 
from NAEP Alliance contractor staff to understand the processes used to develop the web-based 
version of the technical documentation. The second phase of the study involved conducting two 
focus groups, one with outside technical experts and the other with NAEP Alliance contractor 
staff, to review the current web-based documentation and discuss possible changes to improve 
content or accessibility. NCES representatives participated in both of these focus groups. In the 
final phase, we documented study results and presented recommendations for strengthening the 
processes used to develop and maintain web-based technical documentation for the NAEP 
assessments. 



Expert Review 

An expert panel was convened on December 4, 2007 to review the content and 
accessibility of the technical documentation on the web. Participants were identified to cover the 
following areas of expertise: 

• writing technical documentation for large-scale assessments 

• using data and technical documentation from large-scale assessments 

• knowledge of current web technology. 

Representatives from NCES, Alliance Contractors, and HumRRO also participated in the expert 
review as indicated in Table 1. 



Table 1. Meeting Participants 



Organization 


Participants 


Expert Panel 


Brian Gong, Don McEaughlin, William Schafer, Richard Wolfe 


NCES 


Arnie Goldstein, Andy Kolstad, Richard Struense, Bill Tirre 


NESSI 


Kim Gattis, Cadelle Hemphill 


Alliance Contractors 


Debbie Kline (ETS), Paul Harder (Eulcrum IT), Harlan 
Messinger (Eulcrum IT) 


HumRRO 


Monica Gribben (facilitator). Sunny Becker, Hilary Campbell, 
Nikolaos Dimopoulos, Sheila Schultz, Steve Sellman, Eauress 
Wise 
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Review Procedures 

In advance of the meeting, participants were provided with access to the technical 
documentation that was available to the public. Three chapters — Item Development, Instruments, 
and The NAEP Database — were posted online at the time of the meeting (December 4, 2007). In 
addition, password-protected access was provided to additional technical documentation content 
that was in draft status pending review and approval. Discussion during the meeting centered on 
the following, slightly revised, study questions that targeted the expert panelists’ knowledge and 
experience: 

1. Is coverage of the existing topics clear and complete? 

a. How do you or how would you like to use NAEP data? 

b. What information do you need within the documentation to use the NAEP data? 

c. What technical documentation topics do/would you use most often? 

d. How clear is the technical documentation? 

e. How do you rate the ease of navigation of the documentation? 

f. How do you rate the accessibility of the documentation? 

g. How does navigation of documentation on the web impact access to content? 

2. Are there areas where the documentation could be improved or expanded? 

a. Where could the documentation be improved or expanded in terms of the content? 

3. Are there ways in which access to information on specific topics might be improved? 

a. Are there ways navigation could be improved? If so, how? 

b. Are there ways accessibility could be improved? If so, how? 

4. Are there additional topics that might also be addressed by the technical documentation? 

If so, what are they? 

NAEP Technical Documentation as Benchmark 

Participants discussed other technical documentation available on the web - primarily 
reports from state assessment systems and international assessments. Panelists generally agreed 
that most online technical documentation is in the form of downloadable files of paper-based 
reports. Thus, NAEP can provide an educative function in presenting online technical 
documentation that takes full advantage of the benefits of the web (e.g., searching and exporting 
data). 

Recommendations 

Several broad themes emerged from panelists’ discussion across questions. Thus, 
recommendations below are organized by overall themes rather than by responses to individual 
discussion questions. Within each recommendation, detailed content of the panelists’ discussion 
is elaborated upon as appropriate. 

1. Develop a design document with specifications and a timeline for producing, 

reviewing, and publishing the technical documentation on the web (TDW). 
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The expert panelists discussed several issues related to defining the scope of the technical 
documentation and what constitutes “complete” reporting. The content of the TDW is based 
on the printed reports, which are historical in nature with each generally following the format 
of the one before. It is important to maintain some consistency across time for repeat users. 

It was noted that a clear statement of the intended users could facilitate defining or 
refining the scope of the contents. Further, since NCES includes much introductory NAEP 
material on the main web reporting site (www.nationsreportcard.gov), there should be links 
to such material rather than repetition. Thus, a design document should coordinate TDW 
content with other NCES website content. Specifications for what the TDW should be could 
facilitate evaluation against those criteria. 

2. Systematically collect feedback from users. 

Eeedback from users was discussed from two perspectives. Eirst, once a complete set of 
TDW is available, focus groups with “heavy” users of NAEP data and technical 
documentation should be conducted asking some of the same questions about content, 
navigation, and accessibility as discussed by the expert panel. Second, the web pages could 
be used to collect targeted data from users on a regular, ongoing basis. 

The expert panelists did not have a complete set of documentation to review and 
suggested that convening additional focus groups could be informative. Eurther, the experts 
may not have represented the range of users so engaging additional users could provide a 
broader range of perspectives. Eocus group members might be able to address how technical 
documentation relates to other NAEP and NCES content and how the related online content 
might be better integrated. 

Discussion of using the TDW for collecting feedback on the site included a suggestion to 
insert questions for users to rate the usefulness of the page. It was noted survey questions 
would require 0MB approval. In addition, participants noted a specific option tied to the 
particular page within the technical documentation for submitting feedback might be useful. 
Currently, certain feedback submitted through “Contact Us” buttons on the web pages is 
routed to the lES webmaster and then filtered to those responsible for the TDW. This user 
feedback unfortunately does not include information about the source page from which the 
user made the comment. 

3. Provide a summary of TDW content and a guide for using TDW. 

Experts suggested a larger index with a summary of the content. A summarized version 
of the technical documentation would prove useful for providing a sense of the range of 
information included in the site. There was a sense of not being able to fully appreciate what 
the technical documentation includes with the numerous web pages and links to additional 
pages providing subsequent levels of detail. A desire to know the scope of the content was 
expressed to give the user a better idea of how much text would be accessed. This might 
allow the user to assess whether the page would be too much, too little, or appropriate. 
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To aid users, several participants suggested developing a TDW user guide. The guide 
could be a printed manual or a tutorial such as that included with the NAEP Data Explorer. 
Regardless of format, the volume and complexity of the TDW seem to warrant a guide to 
features of the site. 

4. Expand customization of the technical documentation. 

Placing the documentation online allows customization of the content. Expert panelists 
indicated this was an important feature given the different types of users. The “print checked 
items” feature, which allows some customization, should be made more salient. There could 
be some pre-packaged custom versions, such as a collection of the summary-level pages or 
“chapters” that provide in-depth information of a topic. Creating a customized version of the 
technical documentation to meet users’ needs could be accomplished with a “shopping cart” 
tool for users to add sections of the documentation relevant to their needs and interests. 

5. Increase search capahilities within the technical documentation. 

Panelists requested a more robust search capability within the site. Eor web crawlers to 
pick up the TDW content, an expanded high-level version should be available. 

6. Provide data tables in exportable format (Excel). 

Users expressed a need to export data tables into Excel. Eor example, they may want to 
sort data in the table in different ways or to use the data in combination with NAEP results. 
Although the tables within the TDW can be exported from their HTME code. Excel versions 
are not currently available for download. 

Many NAEP data tables include annotations (e.g., daggers) describing the data. Panelists 
noted a need for a glossary of ASCII codes for special characters that appear in tables in the 
TDW to facilitate their use when exporting the tables. 

7. Assess the efficiency of the review process. 

With respect to the review process, it was acknowledged that researchers find the posting 
of technical content useful. To provide the greatest benefit, technical documentation should 
be as current as possible. Panelists suggested a complete but simple set of current 
documentation should be posted in the near-term, with “bells and whistles” added in the 
future. 

Attendees discussed the review process, comparing face-to-face review to online review. 
Participants suggested a review of the tradeoff between efficiency and quality of the steps in 
the review process. Such an investigation might offer insight into how the process could be 
made more efficient using measures of effectiveness (e.g., number and characteristics of 
changes resulting from each step) and resources used (e.g., time to resolution, person-hours 
consumed). Some participants suggested periodic monitoring of the review process. 
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Suggestions for Expanding the Technical Documentation 

There was some discussion of additional content or more in-depth information in certain 
areas. Specific suggestions for additional topics included: 

• increased documentation of full population estimates (FPEs), 

• more information on spiraling, 

• more information on non-response bias, 

• links to technical articles cited in the TDW (e.g., NAEP Validity Study [NVS] reports 
and external citations), 

• a “What’s New?” section, 

• a history or chronology of NAEP, highlighting changes in the assessment, and 

• information on mapping state standards to NAEP standards. 

Participants also suggested reviewing the organization of the data collection and 
weighting chapters. Currently, these chapters are organized by year rather than by topic/year, as 
are other chapters. Another suggested feature was providing a list of similar pages that other users 
accessed when reviewing the same topic. 

Additional or Future Study 

Panelists recommended additional focus groups as part of a future study to refine the 
structure and content of the online technical documentation. Although this investigation is 
beyond the scope of this special study, it may provide future routes for ongoing updates to the 
TDW. Study ideas include: 

• Conduct focus groups with NAEP data users 

Focus group participants could be drawn from grantees conducting secondary 
analyses, NAEP State Coordinators, state testing directors and curriculum specialists, 
representatives from professional organizations (e.g., CCSSO), and media users. 
HumRRO staff suggested participants could be asked to perform specific tasks that would 
require use of the technical documentation during the focus group. Focus group members 
could discuss their success and difficulties in accomplishing the task(s) using the TDW. 
Participants could be asked to rate the usability of different formats of exportable 
information (e.g., data in Excel or html format, text in printer-friendly version). In 
addition, participants could be presented with different page designs to assess 
accessibility and usefulness. The focus group participants could aid in defining the range 
of users and their needs. 



Contractor Review 

A panel of representatives from NAEP Alliance Contractors was convened on December 
4, 2007 to (a) review the accessibility of the technical documentation on the web, (b) review the 
benefits of using the web to disseminate the technical documentation, and (c) discuss the process 
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for developing and disseminating web-based technical documentation. Participants were 



identified by each Contractor. Representatives from NCES, NESSI, and HumRRO also 
participated in the expert review as indicated in Table 2. 

Table 2. Meeting Participants 


Organization 


Participants 


NCES 


Janis Brown, Richard Struense, Bill Tirre 


NESSI 


Kim Gattis 


Alliance Contractors 


Jim Close (Pearson), Debbie Kline (ETS), Paul Harder (Eulcrum 
IT), Harlan Messinger (Eulcrum IT), David Morganstein 
(Westat), Jason Schuknecht (Westat) 


HumRRO 


Monica Gribben (facilitator). Sunny Becker, Hilary Campbell, 
Nikolaos Dimopoulos, Sheila Schultz, Eauress Wise 



Review Procedures 

Debbie Kline (ETS) provided a document outlining the benefits of moving from print to 
online documentation. This was used to guide much of the discussion (see discussion question 
#1). The following questions guided the discussion of the Contractor panel: 

1. Have the benefits of moving NAEP technical documentation to HTME on the web been 
realized? 

a. Is it quicker and easier to update documentation with new information as each 
annual assessment is completed? 

b. Is the review process streamlined since much of the content remains unchanged 
from one assessment to the next? 

c. Are different audiences accommodated efficiently by providing links to additional 
levels of details? 

d. Is searching for specific technical information enhanced? 

e. Is data available in more useful and appropriate formats (e.g., Excel 
spreadsheets)? 

f. Are common topics explained in a single page and linked to as appropriate? 

g. Has information been posted incrementally rather than waiting for all parts of an 
assessment’s documentation to be completed? 

h. Have links to other parts of the NAEP site (e.g., NAEP Data Explorer) been 
incorporated where appropriate? 

i. Has ETS/NAEP web staffs familiarity with NCES’s requirements and 
restrictions facilitated publication of the technical documentation on the web 
(TDW)? 

2. What improvements can be made to the development of online technical documentation? 
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3. How accessible is TDW for those familiar with NAEP and first-time users? 

4. How can the review process be more efficient? 

Overall Reaction to the Technical Documentation on the Web (TDW) 

One expected benefit of placing the documentation online is that it should speed up the 
publication process, which had been increasing over time with the print-based report (growing 
from two years to four years to produce the 1992 and 1996 reports, respectively). However, the 
TDW process has hit snags that have hampered an efficient deployment of the online content. 
There have been changes to the standards and requirements (e.g., statistical standards, web 
format requirements, and policy changes) that have added time to the review cycle as well as the 
creation of the web pages. Participants expressed frustration with the delays in publishing the 
technical documentation and hope that once the initial TDW is complete (for NAEP 2000), 
subsequent years’ documentation will be posted with less delay. Contractors and NCES staff 
made several recommendations for improving the TDW process. 

Recommendations 

Several themes emerged from the discussion between the Alliance contractors, NCES, and 
NESSI staff. The discussion allowed participants to ask questions of ETS, Eulcrum IT, and 
NCES representatives as well as provide suggestions for changes to the TDW process. 
Recommendations below are organized by overall themes rather than by responses to individual 
discussion questions. Within each recommendation, detailed content of the panelists’ discussion 
is elaborated upon as appropriate. 

1. Develop and follow a design plan. 

Participants noted that there is no design plan for the technical documentation. Although 
each contract includes language requiring the contractor to produce technical documentation, 
there are no specifications for the content, no formal written plan to integrate the 
documentation, and no timeline (see recommendation #2). Contractors believe that a design 
plan could guide the process by providing specifications for writing and reviewing the 
documentation as well as detailing requirements for posting the material online. As part of 
the design plan, content and audience could be specified. Current NAEP products (e.g., 
scoring memorandum) may be identified as the basis for the content of the technical 
documentation to minimize the time needed to produce the content. With a written plan, 
contractors believe that the process should be more efficient. 

2. Initiate a schedule for release of technical documentation. 

The contractors discussed several issues related to the publication timeline. Primary 
among the issues was lack of a schedule for releasing the documentation. Without a schedule, 
delays more easily occur during one or more steps of the process: (1) producing the content, 
(2) reviewing the documentation, or (3) publishing it in online format. Other issues were 
turnover in reviewers, changes to the review process, and TDW reviews being placed on the 
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“back burner.” These have resulted in additional time needed for approval of the TDW. In 
some cases, content was re-reviewed because new members joined a review committee. In 
other cases, TDW pages were submitted for review but the review process seemed to stall 
mid-review. Further, there is no design plan (see recommendation #1) or guide for reviewers 
so comments from different reviewers can be contradictory. Reconciling disparate comments 
has the potential to increase the time needed to produce an approved version of the 
documentation. Suggestions included adding the technical documentation to the “Big 
Schedule” and/or legislating timelines for the documentation. Participants believe that a 
schedule for TDW would help to reduce delays at all points in the process. 

3. Emphasize the search capability within the TDW (vs. the NAEP weh pages). 

Echoing feedback from the expert panelists on search capability within the TDW, the 
contractors and NCES staff discussed different levels of search capability (NCES, NAEP, 
and TDW sites). There was discussion of highlighting the search within the TDW versus 
searching at more general levels such as NAEP and NCES. NCES and NAEP web standards, 
including restrictions on pop-up windows, limit the technical options for offering a search 
capability. In general, the ability to search is an added benefit of moving to online 
documentation (the print technical documentation did not have an index). Additionally, an 
updated search tool is planned. 

4. Document review and revision decisions. 

Contractors discussed the review and revision cycle. There is concern revisions are being 
undone during the review process when material is reviewed multiple times. Turnover on 
review committees has led to re-review of content that has not been approved because 
extended time has passed between revisions and reviews. In some cases, changes requested 
in early reviews have been removed during re-reviews conducted at the same level (albeit 
with new members on the review committee). To make the review process more efficient, 
contractors suggest full documentation of review comments and subsequent decisions of how 
to revise the content. Contractors would like decisions to change, correct, or revise the 
content to be locked at each stage of the review. 

One of the expected benefits of TDW was that some content would not change from 
assessment to assessment. As a result, the review process should be streamlined. The 
suggestion to lock pages should span technical documentation across assessments. Content 
that does not change should not be subject to review in future years. 

Contractors discussed the review process, comparing the collective review session to the 
review and revise cycle. The general feeling was that the review and revise cycle is not time- 
efficient given the extended time that reviews have been taking. 

5. Collect data on the review process. 

Contractors suggested collecting data on the timing and/or steps in the review process to 
provide information on where bottlenecks and delays are occurring. Currently, the original 
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schedule is replaced with revised dates when changes are made so that originally proposed 
milestone dates are not saved. Therefore, there is no comprehensive record of the length of 
time reviews are taking/have taken. By maintaining an historical record of the original review 
timeline and the actual review dates, these data could be used to inform suggestions to 
modify the review process based on relevant data. In addition, collecting data on the review 
process would allow periodic feedback on the efficiency of the process. Further, changes to 
the process could be evaluated quantitatively. 

6. Collect feedback from users via surveys. 

Soliciting feedback from users of the technical documentation could inform the authors 
and others involved in the publication of the information. Users could be asked to comment 
on the level of detail and usefulness of the content. Users’ comments might be used to 
provide guidance for authors and reviewers concerning the appropriate levels of detail for 
different audiences. 

7. Limit changes in standards, requirements, and formats. 

Changes in standards require updating approximately 1,500 pages of TDW content to 
date. Although in some cases it can be done automatically, it affects timelines nonetheless. 
Pages have to be changed to conform to new standards and reviewed for compliance with the 
new standards. Thus, changes in standards can add an additional update/review cycle to the 
TDW process. Recognizing that standards change, contractors suggested ways to mitigate the 
impact on publishing the technical documentation. Participants suggested that the 
implications of complying with new standards should be considered when changes are being 
considered. NCES might consider allowing TDW to proceed under old standards once 
reviews have begun or once a certain point in the process has been reached. In essence, there 
would not be as much revision required as a result of changes in standards and/or 
requirements. Further, advance notification of changes to standards, requirements, formats, 
etc. should be communicated to the NCES staff involved in producing the technical 
documentation. 

8. Develop a Users’ Guide for the TDW. 

Participants suggested a Users’ Guide be developed to introduce the content and tools 
included in the TDW. The Guide would provide an overview of the site and other related 
sites (e.g., NAEP, NCES, and NAEP Data Explorer). It would include sections on how to get 
the most of the online technical documentation, such as how to: search, download tables, and 
print selected sections. 

9. Collect information about users. 

Participants suggested Webtrends should be used to collect information about individuals 
accessing the TDW. The information could be used to provide a profile(s) of the audience to 
authors and other relevant parties (e.g., reviewers). Such information should be collected and 
disseminated periodically. 
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Additional or Future Studies 

Panelists recommended several further or future studies that will assist in understanding 
the TDW process and produce some guidelines for best practices. Although these investigations 
are beyond the scope of this special study, they may provide future changes to the process for 
developing a more efficient workflow. Study ideas include: 

• Conduct a cost-benefit analysis of print-based and web-based technical 
documentation 

A list of expected benefits to online documentation was discussed during the 
session. A cost-benefit analysis could demonstrate whether the benefits have been 
realized and if they result in a cost savings. Benefits that might be included in the 
analysis are (a) quicker and easier updating, (b) streamlined review process, (c) 
accommodating different audiences, (d) search capabilities, (e) downloadable data, (f) 
linking to common information (compared to repeating it), (g) publishing information 
incrementally, as available, and (h) linking to other NAEP sites. 

• Explore differences between page navigation models for web-based technical 
documentation 

The NAEP TDW site consists of approximately 1,500 pages of technical content. 
It is both broad and deep. To assist users, the NAEP TDW site uses a standard crumb trail 
navigation model to provide a “map” to where the user is within the site. The standard 
crumb trail is a navigation tool that allows a user to see where the current page is in 
relation to the Web site's hierarchy. In the NAEP TDW site this is not necessarily the 
path that the user took to get to the page, which requires a dynamic crumb trail navigation 
model. Dynamic crumb trails show the pages visited by the user as he/she works his/her 
way through the site. As with the standard crumb trail, there are benefits and 
disadvantages to using the dynamic navigation tool. 

Research suggests that crumb trails are most useful for hierarchically organized 
sites (e.g., NAEP TDW) and they tend to be used more by expert users than novices. 
However, content for some sections within the NAEP technical documentation does not 
fit the standard crumb trail model well. In addition, this model seems to lose some of the 
benefits of web-based documentation (e.g., content can generally have only one pathway, 
therefore content with application to multiple places on a web site might have to be 
duplicated to stay within the crumb trail hierarchy). To achieve the benefits of placing the 
documentation online, it might be instructive to examine the differences between page 
navigation models with respect to the content and structure of the technical 
documentation. Such a comparison would need to consider the web requirements and 
restrictions of NCES web sites/pages. 
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Summary 

HumRRO staff reviewed the recommendations from each of the two panel meetings. In 
this last section of our report, we combine and summarize key recommendations for further work 
on the web-based documentation. Panelists provided recommendations focusing on (a) the 
content of the documentation, (b) the development process, and (c) areas for further research. 

The sources of the recommendations are noted in parentheses. Suggestions made by both panels 
are listed first within each category. 

Recommendations Concerning the Content of the Web-Based Technical Documentation 

1. Provide a summary of the content and develop a User’s Guide. (Both Panels) 

2. Increase/emphasize search capabilities within the technical documentation. (Both Panels) 

3. Expand customization of the technical documentation. (Expert Panel) 

4. Provide data tables in exportable format (Excel). (Expert Panel) 

Recommendations Concerning the Development Process 

5. Develop a design document/plan with specifications for producing, reviewing, and 
publishing the technical documentation on the web. (Both Panels) 

6. Initiate a schedule/timeline for release of technical documentation. (Both Panels) 

7. Collect data to assess the efficiency of the review process. (Both Panels) 

8. Document review and revision decisions. (Contractor Panel) 

9. Eimit changes in standards, requirements, and formats. (Contractor Panel) 

Recommendations for Further Research 

10. Systematically collect feedback from users - via focus groups or surveys. (Both Panels) 

11. Collect information about users (using Webtrends). (Contractor Panel) 

12. Conduct a cost-benefit analysis of print-based and web-based technical documentation. 
(Contractor Panel) 

13. Explore differences between page navigation models for web-based technical 
documentation. (Contractor Panel) 
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